Evaluating Language Models for Knowledge Base Completion

Structured knowledge bases (KBs) are a foundation of many intelligent applications, yet notoriously incomplete. Language models (LMs) have recently been proposed for unsupervised base completion (KBC), yet, despite encouraging initial results, questions regarding their suitability remain open. Existing evaluations often fall short because they only evaluate on popular subjects, or sample already existing facts from KBs. In this work, we introduce novel, more challenging benchmark dataset, and methodology tailored realistic assessment the KBC potential LMs. For automated assessment, curate dataset called WD-Known, which provides an unbiased random Wikidata, containing over 3.9 million facts. second step, perform human evaluation predictions that not in KB, as real insights into added value Our key finding is biases conception previous benchmarks lead to systematic overestimate LM performance KBC. However, our results also reveal strong areas We could, example, significant Wikidata relations nativeLanguage, by factor $$\sim $$ 21 (from 260k 5.8M) at $$82\%$$ precision, citizenOf 0.3 4.2M 5.3M) 90% precision. Moreover, find LMs possess surprisingly generalization capabilities: even where most were directly observed training, prediction quality can be high. open-source code. ( https://github.com/bveseli/LMsForKBC ).